Skip to content

Strip credentials from origin URL in deploy telemetry - #7331

Draft
pietern wants to merge 1 commit into
mainfrom
pietern/f01-6842-telemetry-origin-url
Draft

pietern wants to merge 1 commit into
mainfrom
pietern/f01-6842-telemetry-origin-url

Conversation

@pietern

@pietern pietern commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Strip credentials from the git origin URL before it goes into deploy telemetry. bundleGitInfo assumed the URL was already stripped, but only the local .git route does that; a URL from bundle.git.origin_url or the workspace API went through as written. The stripping in Repository.OriginUrl() moves into git.StripURLCredentials, which telemetry now uses too.

DMS and deploy metadata still receive the value as written; left for a follow-up.

Why

#6842 added the origin URL to deploy telemetry, assuming its credentials were already stripped.

Tests

  • TestStripURLCredentials in libs/git and a wiring test in bundle/phases.
  • New acceptance test bundle/telemetry/deploy-git-origin-url-credentials shows the token stripped.

This pull request and its description were written by Isaac.

@github-actions github-actions Bot added the DABs DABs related issues label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 23e03dcbe | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (fa7dc9b) latest (v1.26.0)
TestAccept/bundle/telemetry/deploy-git-origin-url-credentials/DMS= ✅ ❌ ❌
TestAccept/bundle/telemetry/deploy-git-origin-url-credentials/DMS=: branch ✅ | main (fa7dc9b) ❌ | latest (v1.26.0) ❌

main (fa7dc9b):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...\nFiles: 1 uploaded, 0 deleted\nResources: 0 created, 0 changed, 0 deleted, 0 unchanged\n\n>>> cat [OUTPUT_DIR]/out.requests.txt\n{\n  \"git\": {\n    \"origin_url\": \"https://github.com/databricks/cli\"\n  }\n}\n"
        	            	actual  : "\n>>> [CLI] bundle deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...\nFiles: 1 uploaded, 0 deleted\nResources: 0 created, 0 changed, 0 deleted, 0 unchanged\n\n>>> cat [OUTPUT_DIR]/out.requests.txt\n{\n  \"git\": {\n    \"origin_url\": \"https://user:token@github.com/databricks/cli\"\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,3 @@
        	            	   "git": {
        	            	-    "origin_url": "https://github.com/databricks/cli"
        	            	+    "origin_url": "https://user:token@github.com/databricks/cli"
        	            	   }
        	Test:       	TestAccept/bundle/telemetry/deploy-git-origin-url-credentials/DMS=
        	Messages:   	bundle/telemetry/deploy-git-origin-url-credentials/output.txt vs /tmp/TestAcceptbundletelemetrydeploy-git-origin-url-credentialsDM4192663553/002/output.txt
FAIL

latest (v1.26.0): same output as main (fa7dc9b)

Comment thread libs/localenv/uv.go Resolved
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 23e03dc

Run: 43847204035

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 4:49
💚​ aws-windows-latest-4core-16gb 1 272 14 5:40
💚​ azure linux-2core-8gb 1 269 16 4:33
💚​ azure-windows-latest-4core-16gb 1 271 14 4:43
💚​ gcp linux-2core-8gb 1 270 16 4:22
💚​ gcp-windows-latest-4core-16gb 1 272 14 6:11
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:51 gcp-windows-latest-4core-16gb TestAccept
3:31 aws-windows-latest-4core-16gb TestAccept
3:15 azure-windows-latest-4core-16gb TestAccept

@pietern
pietern force-pushed the pietern/f01-6842-telemetry-origin-url branch from e28ea52 to 35adebc Compare November 4, 2026 12:26
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 11, 2026
## Changes

Add a `pr-review` agent skill under `.agents/skills/`. It reviews a
branch (or a given commit) against its merge base with main and prints
inline comments, a review body, and the evidence and worksheet behind
them. It never posts.

The skill works through fixed passes: correctness, duplicated
mechanisms, fit with existing extension points, new error paths on
absent input, sibling routes (create vs update, deploy vs destroy),
user-facing names, stored state over time, doc comments, tests, and
user-facing text. It reads the repo's `.agents/rules/` for testing level
and comment conventions.

## Why

We want agent reviews to catch the kind of issues maintainers raise in
this repo, not just generic bugs, and to know how often they do. The
skill was written and tuned against past reviews on this repo rather
than from a checklist.

## Tests

**Backtest against human reviews.** Merged PRs, each frozen at the
commit that received the most human review comments, were reviewed by
the skill with only read-only git access (no PR description, no GitHub),
and an LLM judge checked whether it raised the same issue as each
substantive human comment (match 1, partial 0.5).

- 24 PRs, 96 substantive human comments, split into a tuning set (12
PRs) and a held-out set (12 PRs) whose comments were not read while
writing the skill.
- On held-out PRs the skill reproduces 41% of substantive comments, with
5.4 comments per PR. An earlier version that asked for three to eight
comments per review reproduced 34% (range 30–38% over two runs) with 5.1
comments per PR, and a first generic version 31%. The largest gains are
on PRs with many human comments, where the comment budget used to cut
real findings.
- Comments on tests and acceptance output are caught least often (about
28% vs 41% for source, measured on the earlier version).

**Sweep of recently merged PRs.** The skill reviewed the 59
human-authored PRs merged into main between 2026-10-27 and 2026-11-03,
at their final commit. It made 200 findings (about 3.4 per PR); 34 had
already been raised on the PR. Each finding was then checked against the
code on its own by a separate model told to try to refute it; that
checker refuted all five deliberately false claims planted to test it.
193 findings were confirmed, 6 refuted and 1 left inconclusive. Of the
confirmed ones not raised on the PR, 12 were bugs still present on main.
Fixing them showed 2 to be working as expected; the other 10 are draft
PRs databricks#7331–databricks#7340, each naming the PR that introduced the issue. The
refuted findings were mostly behavior that the code or a test already
documented as intended.

The sweep was run with the earlier version that asked for three to eight
comments per review.

Each review takes 2–5 minutes with Opus. The backtest harness and
fixtures are kept outside this repo so future runs can't read the
expected comments.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
bundleGitInfo assumed load_git_details had already stripped userinfo from
bundle.git.origin_url, but that only happens on the local .git route. A URL
set in databricks.yml, or returned by the workspace API for a Git folder,
was sent to telemetry as written, including any embedded token.

Strip userinfo when building the telemetry git info so every route is
covered. Extract the stripping from Repository.OriginUrl into
git.StripURLCredentials and use it for telemetry too.

Co-authored-by: Isaac <no-reply@databricks.com>
@pietern
pietern force-pushed the pietern/f01-6842-telemetry-origin-url branch from 35adebc to 23e03dc Compare November 16, 2026 10:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants